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This paper studies a diffusion model that arises as the limit of a 
queueing system scheduling problem in the asymptotic heavy traffic 
regime of Halfin and Whitt. The queueing system consists of sev¬ 
eral customer classes and many servers working in parallel, grouped 
in several stations. Servers in different stations offer service to cus¬ 
tomers of each class at possibly different rates. The control corre¬ 
sponds to selecting what customer class each server serves at each 
time. The diffusion control problem does not seem to have explicit 
solutions and therefore a characterization of optimal solutions via the 
Hamilton-Jacobi-Bellman equation is addressed. Our main result is 
the existence and uniqueness of solutions of the equation. Since the 
model is set on an unbounded domain and the cost per unit time is 
unbounded, the analysis requires estimates on the state process that 
are subexponential in the time variable. In establishing these esti¬ 
mates, a key role is played by an integral formula that relates queue 
length and idle time processes, which may be of independent interest. 

1. Introduction. We consider optimal scheduling control for a class of 
queueing systems that operate in heavy traffic, in the sense that the load 
on the system is nearly equal to its capacity. As often occurs, exact analysis 
of the control problem is unavailable and an asymptotic approach is taken, 
where a parametrization of the model is introduced and a diffusion control 
problem is obtained in the limit. The parametrization that has been more 
common in research papers on related problems (referred to here as conven¬ 
tional heavy traffic) is one where arrival and service rates are both scaled up 
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in a way that the system operates near full capacity. Recently, several papers 
have studied a different parametrization, proposed by Halfin and Whitt [10], 
where increase of arrivals is balanced by scaling up the number of (identical) 
servers in each service station, while keeping the service time distribution 
of the individual servers fixed. In the limit as the parameter grows without 
bound, conventional heavy traffic typically gives rise to diffusion processes 
in the orthant with reflection on the boundary, whereas in the Halfin-Whitt 
(HW) regime the diffusion takes values in the Euclidean space. This pa¬ 
per focuses on the diffusion model that corresponds to the queueing system 
introduced below, operating in the HW regime. 

The queueing system has a fixed number of customer classes and many ex¬ 
ponential servers grouped in a fixed number of stations. Only some stations 
can offer service to each class, and the service rates depend on the class and 
on the station. Also, customers not being served may abandon the system 
(see Figure 1). Scheduling (and routing) of jobs in the queueing system is 
regarded as control. As cost one considers an expected discounted cumula¬ 
tive function of performance criteria such as queue lengths, number of idle 
servers or number of customers of each class present at each station. The 
system is parametrized so that the arrival rates and the number of servers 
at each station are nearly proportional to a large parameter n, while service 
and abandonment rates are nearly constant. For motivation on the model 
and on this asymptotic regime, see [4] and [12]. 

In the scaling limit and under appropriate assumptions, one obtains a 
diffusion model whose ingredients are denoted by X,Y,Z,'^ and W. Let I 
and J' be index sets for customer classes and service stations, respectively. 
Assume that a sequence of systems is given where, in the nth system, the 
number of servers at each station is proportional to n. For each system n, 
denote by X^, i I, the number of customers of class i present in the system. 
Then X stands for the (formal) weak limit of the processes {X^ — nx*), 



Fig. 1. System with four customer classes (buffers) and three server types (service sta¬ 
tions). 
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where x* are constants that come from a corresponding “static fluid model” 
(see [2] for full details on the fluid model as well as on the derivation of 
the diffusion model from the queueing model). Note that because of the 
centering, Xi assume both positive and negative values. In an analogous 
fashion the processes Yi, i gT, and Zj, j G J, correspond to queue length 
of class i and, respectively, the number of idle servers at station j, and 
^ij corresponds to the number of class-i customers in service at station j. 
Finally, Wi are Brownian motions that represent the effect of fluctuations 
in arrival and service times. With and [lij standing for the abandonment 
rate of class-i customers and, respectively, service rate of class-i customers 
at station j, the diffusion model reads 


(1) 

Xi{t) = Xi + Wi{t) - V fiij / ^ij(s) 

ds — 6i / Yi{s) ds 


jsj 40 

Jo 

(2) 




j&j 



(3) 




i&I 



(4) 

Yi > 0, Zj > 0, 

i G J,j G J. 

(5) 

min 


= 0. 


ugx jGj" J 


Denote by T the graph that has a node per each class and each station, 
and an edge that joins the class and the station if and only if customers of 
the class can be served at the station. It is assumed in this paper that T is 
a tree (see Section 2 for discussion on various aspects of this assumption). 
It is then possible to encode equations (l)-(5) in a single equation of the 
form X(t) =x + rW{t) + /q 6(X(s), C/(s)) ds, where 17 is a control process 
that takes values in a compact space and VF is a standard Brownian motion. 
Also, rescaling the cost associated with the queueing system appropriately 
results in a cost of the form e~^^L(Xt,Ut)dt for the diffusion model. 

The main result of this paper is characterization of the diffusion control 
problem’s value as the unique solution to the associated Hamilton-Jacobi- 
Bellman (HJB) equation. Such problems for diffusion in an unbounded do¬ 
main are well understood when either the “running cost” L is bounded [5, 7] 
or the drift b is bounded (as follows from the results of [13], e.g.). The diffi¬ 
culty in the current model stems from the fact that the domain as well as the 
functions L and b are not necessarily bounded. The question then requires 
deeper understanding of the model and, in particular, estimates on moments 
of the process that are subexponential in the time variable. Our results ap¬ 
ply when either one of the following conditions holds (further conditions on 
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abandonment rates are assnmed in each case; see Section 2): 1. Service rates 
are either class- or station-dependent; 2. T satisfies diam(T) < 3; 3. Running 
cost L{X,U) is comparable to ||^||™, some m > 1; 4. L is bounded. 

The large time estimates needed for the first three cases of the main 
result use three different strategies. One of our basic tools is an integral 
equation (51) that expresses a relationship directly between Y, Z and W, 
not involving X and T. The special form this equation takes when /Xjj depend 
only on i or only on j makes it possible to get the required estimates on X 
in case 1 of the main result. Equation (51) may be of independent interest 
since it can be seen to express a relationship between the data W, the control 
process U and the one-dimensional process alone (cf. Section 4). 

The integral equation is used also in case 2 of the main result, along with 
a certain property of the system (l)-(5). This property, which we call the 
nonidling property, should be understood as one of the system of equations 
rather than the stochastic processes, because it does not regard W as a 
Brownian motion, but as a generic function: 

If the system starts with Xi > 0 and 1 1 —> Wi(t) are strictly increasing, 
then Zj{t) = {), j G J, f >0. 

For trees of diameter 3 or less we can show that the property holds and 
that it implies moment estimates that are polynomial in time. Heuristically 
this property expresses that servers do not idle when there is large enough 
amount of work. Viewed this way it appears to be a basic question on the 
model and one would like to understand in what generality it holds (e.g., 
for more general trees), irrespective of the goals of the current work. 

In case 3 of the main result we assume that the function L is comparable 
to a power of the norm of the state process X. This assumption is not the 
most natural in the context of queueing, since, for example, cost functions 
that depend on queue lengths cannot be treated. However, it can be nseful 
if one is interested in stabilizing the dynamical system about a nominal 
model, since the cost penalizes deviations from the static fluid model. On 
the technical side, penalizing deviations from a nominal model simplifies the 
problem in that moment estimates are required under one particular control 
rather than under all controls, and as a result we can treat the model at full 
generality, as far as the tree structure and the service rates are concerned. 
Finally, in case 4, where L is assumed to be bounded, estimates on X are 
not needed. 

Apart from their own contribution the results of this paper are a first step 
toward identifying schednling policies for the queueing system that are, in an 
appropriate sense, asymptotically optimal [2]. As in [4], such policies can be 
derived from the solution to the diffnsion control problem. In addition, such 
asymptotic analysis jnstifies the relationship between the queneing system 
and the diffusion model studied here. 
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Recent results on the HW regime include the following. Puhalskii and 
Reiman [17] extended the work of Halfin and Whitt to multiple customer 
classes, priorities and phase-type service distribution. Mandelbaum, Massey 
and Reiman [15] established functional law of large numbers and central 
limit theorems for a wide class of Markovian systems in the HW regime. 
Armony and Maglaras [1] modeled and analyzed rational customers in equi¬ 
librium, and Garnett, Mandelbaum and Reiman [8] studied models with 
abandonments from the queue. Papers where a control theoretic approach 
was taken to study queueing systems in this regime are few. The diffusion 
control problem associated with scheduling jobs in a system with multiple 
customer classes and a single service station was analyzed by Harrison and 
Zeevi [12]. In a similar setting, Atar, Mandelbaum and Reiman [4] estab¬ 
lished asymptotic optimality of scheduling policies for the queueing system 
derived from the diffusion model’s HJB equation. A special case where ex¬ 
plicit, pathwise solutions to the diffusion control problem are available ap¬ 
peared in [3]. Finally, with regard to the conventional heavy traffic analysis 
of systems analogous to the one studied in the current paper, we mention 
Harrison and Lopez [11], where the corresponding Brownian control problem 
is identified and solved, Williams [18], where a dynamic threshold scheduling 
policy is proposed for the queueing system, and Mandelbaum and Stolyar 
[16], where asymptotic optimality of a simple scheduling policy is proved for 
convex delay costs. 

The organization of the paper is as follows. Section 2 describes the queue¬ 
ing system and the diffusion model and states the main result. Section 3 
shows how estimates on ||Ai|| reduce to estimates on \ Section 4 de¬ 

velops the integral equation (51) and establishes moment estimates in case 
1 of the main result. Section 5 studies the nonidling property for trees of 
diameter 3 and establishes case 2 of the main result. Section 6 treats case 
3 and summarizes the estimates in all cases. Based on the estimates of Sec¬ 
tions 3-6, the proof of the main result is similar to an analogous treatment 
in [4], but for completeness we provide it in the Appendix. 

Notation. For /: [0, oo) ^ M, let 3f = /q /. For a vector x, let ||x|| = X) 1^ 
For two column vectors v and u, v ■ u denotes their scalar product. The 
symbols denote the coordinate unit vectors and e = (1,...,!)'. The di¬ 
mension of e may change from one expression to another, and, for example, 
e ■ a = e ■ b makes sense even if a and b are of different dimension. The 
symbol (resp. C'™') denotes the class of functions on for which 

all derivatives up to order m are Holder continuous uniformly on com¬ 
pacts (resp. continuous); Cpoi denotes the class of continuous functions / 
on M^, for which there are c, r such that \f{x) \ < c(l -|- ||®||^), x G Let 
~ ^poi and C^i = Cpoi n C"", and let be the class of 

nonnegative functions in C^j. The symbol Cb denotes the class of bounded 
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continuous functions and C™ = D Cb- Let M+ = [0, oo). If X is a process 
or a function on M+, ||X||j = supQ<^<^ ||X(s)||, and if X takes real values, 
|X|j = supo<s<jX(s)|. The symbols X{t) and Xt are used interchange¬ 
ably. The symbols ci, c'^, C 2 ,C 2 ,... denote deterministic positive constants 
that may have different values in the proof of different results. 

2. The queueing system and the diffusion. We start with an example 
that demonstrates how the HW scaling is performed on a simple queueing 
system. The queueing system, parametrized by n € N, has a single customer 
class with renewal arrivals at rate A” and n servers, each having exponential 
service time distribution of rate While a customer is not in service, it 
abandons the system at rate 0”. Let X"'(t), Y'^{t) and denote the 

total number of customers, the number of customers not being served and 
the number of servers that are idle at time t, respectively. Clearly 

( 6 ) X^ + Z^ = n + Y^. 

Assume that the system operates under work conservation, in the sense that 
yn /\ = 0. Then T" = (X"’ — n)+ and Z” = (X" — n)~. The parameters 

scale so that — A) ^ A, — /i) —> /i and —> 6. The system 

is assumed to be critically loaded, in the sense that 

(7) X = ^^. 

Denote X^{t) = n-^/^{X^{t)-n),Y^{t) = n-^/‘^Y^{t) and Z^{t). 

Assuming that the interarrival times have finite second moment and that 
X”(0) —> X, X” converges weakly to a diffusion X that solves 

(8) X{t) = X + rW{t) + [ {X — fl + fJ-X{s)~ — 9X{s)^) ds, 

Jo 

where W is a standard Brownian motion and r is a constant that depends 
on the first two moments of the interarrival time and on A (in the absence 
of abandonment, the result is due to [10]; see [8] for a treatment of aban¬ 
donment). Denoting by Y and Z the weak limit of T” and Z”', respectively, 
note that (8) can be rewritten as 

(9) X{t) = X + rW{t) + [ {X —fi +fJ-Z{s) — 9Y{s))ds, 

Jo 

(10) X = y-Z, YAZ = 0. 

We repeat that some nonexponential service distributions were treated in 
[17]; however, the diffusion limit turns out to be more complicated and that 
approach is not taken here. 

The diffusion model studied in this paper corresponds to a queueing sys¬ 
tem with several classes and stations. The queueing system and the corre¬ 
sponding diffusion model are introduced below. Since in the current paper 
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we focus on the diffusion model, we present it here without attempting to 
justify its relationship to the queueing system, and we demonstrate only how 
it is analogous to the simple model (8)-(10). Full details on this relation¬ 
ship are deferred to [2], The queueing system has I customer classes and J 
service stations (see Figure 1). At each station there are many independent 
servers of the same type. Each customer requires service only once and can 
be served indifferently by any server at the same station, but possibly at 
different rates at different stations. Only some stations can offer service to 
each class. Label the classes (and corresponding buffers) as !,...,/ and the 
types (and corresponding stations) as I -|- 1,..., / J, and set 

J={1,.J = {/ + l,...,/ + J}. 

The structure of the system can be encoded in a graph. A pair {i,j) G T x ^ 
is called an activity if customers of class i can be served at station j. It is 
assumed that whether (i, j) is an activity does not depend on n. Let T denote 
the graph with vertex set {1,2,..., I + J} = T [J J: A node is associated 
with each buffer and each station. Edges of T are between elements i G T 
and j ^ J such that (z,j) is an activity. Write i ^ j if (i,j) is an activity. 
Denote the edge set for the graph by 


£ = {{i,j) el X J:ir~. j}. 


Eor j G J, let A"" be the number of servers at station j. Let A”(t), Y^{t) 
and Zj'{t) denote the total number of class-i customers in the system, the 
number of class-i customers in the queue, and the number of idle servers 
in station j at time t, respectively. Einally, let 'h(^ (t) denote the number of 
class-i customers in service at station j at time t (note that = 0, z / j). 
In vector-matrix notation, set A” = (A”)jex, E"' = Z"’ = 

and T” = ('L^)jgj^i£j-. Straightforward relationships are expressed by the 
equations 

(11) zGJ, 

j&J 

( 12 ) 

iex 

(13) Yp{t),z]{t)>Q, iei,jej,t>o. 


Arrivals of class-i customers occur at rate abandonment from queue i is 
at rate 6f and, for i^ j, service of a class-z customer at station j is at rate 
Assume 


(14) n^/^(rz - Ai) ^ Aj, - mj) ^ jlij, 

(15) n^/2(re"^A”-zyj) ^0. 
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Above, Xi,h'j G (0, oo), 9i G [0,oo), A G M and for i ~ j, fiij G (0, oo) and 
fiij G M. It is convenient to set throughout /ijj = fiij = 0 for i 9 ^ j. In analogy 
with (7) in the simple example, we invoke a condition that expresses that 
the system is critically loaded. The condition involves the “first order” pa¬ 
rameters Xi^Vi and and certain constants x* and that represent a 
static fluid model (see details in [2]). The processes are rescaled as 

(16) Xr{t) = n-^/\xr{t)-nx:), 

(17) y-(t) = n-V2y.-(t), z^{t) = 

Assuming A”’(0) —> x and considering X, Y, Z and d/ as formal limits of 
A"", y”, Z^ and T”, we expect in analogy with (9), 

(18) Xi{t) = Xi + Wi{t) -iJ.ij [ '^ij{s)ds - 9i [ Yi{s)ds, iel, 

jGj -^0 

holds, where Wi{t) = rjlTj(t) + tit^ IT is a standard Brownian motion, r* G 
(0, 00 ) and 4 G M are constants, ^ij = 0 for i 9 ^ j, and, in view of (11)-(13), 


(19) 

^ = A, - T, 

i G T, 


3&J 


(20) 


j^J, 


iGJ 


(21) 

y) > 0, Zj 

>0, i£l,jej. 


For reasons explained in [2], the work conservation condition is replaced by 
the condition 

(22) e-yAe-Z = 0. 

The diffusion model is now described by (18)-(22). To view the model in a 
control theoretic setting, regard 'h as a control process and A as a controlled 
diffusion. Then (18) describes the dynamics of A, and (19)-(22) serve to 
define the “auxiliary” processes Y and Z and to set constraints on Note 
that the constraints on 'I' involve the process A. 

While relationships (18)-(22) are, in a sense, analogous to (9)-(10) (al¬ 
though obviously there is no control process in the simple model), we would 
like also to have a relationship analogous to ( 8 ). More precisely, we seek to 
describe the model in the convenient form 

X{t) = x + rW{t)+ [\{X{s),U{s))ds 

Jo 


(23) 
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with r = diag(rj)jgi, appropriate function b and control process U, and, in 
particular, without having to impose constraints on U that involve X. The 
assumption below is useful in this development; however, as discussed at the 
end of this section, the prime reason for imposing it is different. 

Assumption 1 (Treelike). The graph T is a tree. 

Proposition A.2 in the Appendix shows that under Assumption 1, (19) 
and (20) are equivalent to 

(24) T = G(A-y,-Z), 

G being a linear map from {{a, j3) G ^ a* = ^/3j} to We proceed 

to derive (23) under the treelike assumption. Note first that by (19) and 
(20), e-X = e- Y — e-Z, and thus by (21) and (22), e ■ Y = (e ■ X)'^ and 
e ■ Z = {e ■ X)~. Hence Y can be represented as 

(25) Yi{t) = {e-X{t))^Ui{t), 

where Ui{t) > 0 and e • u{t) = 1. Similarly, 

(26) Zj{t) = {e-X{t))~Vj{t), 

where Vj{t) > 0 and e • v{t) = 1. Consider U := {u,v) as a control process 
that takes values in 

U := {{u, v) G ■Ui,Vj >0, e - u = e - v = 1}. 

By (24), 

T = G{X, U) := G{X - (e • A)+n, -(e • X)-v). 

Let 

(27) h{X, U) = -J2 f^ijG{X, U),j - Oiie ■ X)+u, + ii 

and write b = ibi)i^x- We see that (18) can be written as (23). 

Definition 1 (Admissible systems and controlled processes). 

(i) We call 

7r = in,F,{Ft),P,U,W) 

an admissible system and say that t/ is a control associated with vr if 
{Q, F, (Ft), P) is a complete filtered probability space, U is an (Ft) pro¬ 
gressively measurable U-valued process and IT is a standard /-dimensional 
(Ft) Brownian motion. 
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(ii) We say that X is a controlled process associated with initial data 
X G and an admissible system vr = (O, F, {Ft),P, U, W) if X is a continu¬ 
ous, (Ft)-adapted process such that P-a.s., /q |6(X(s), ?7(s))| ds < oo and 

(28) X{t) = X + rW{t) + [ b{X{s),U{s))ds, 0<t<oo. 

Jo 

As stated in Proposition A.l in the Appendix, there is a unique controlled 
process X associated with any x and vr. With an abuse of notation we 
sometimes denote the dependence on x and vr by writing P^ in place of P 
and E'^ in place of E. Denote by IT the class of all admissible systems. 

Let a constant 7 > 0 and a function L be given, and consider the cost 

poo 

C{x,7r) = E^ / e-'^^L{X{t),U{t))dt, xeM^vreH. 

Jo 

Our assumption on L is as follows. 

Assumption 2. (i) We have L{x, U)>0 and (x, [/) G x U. 

(ii) The mapping (x, U) 1 —> L(x, U) is continuous. 

(iii) There is ^ G (0,1) such that for any compact A C 

|L(x,[/)-L(y,[/)|<c||x-yr 

holds for [/ G U and x,y € A, where c depends only on A. 

(iv) There are constants cl > 0 and mi > 1 such that L{x,U) < cl(1 -|- 
||x||™^), C/gU, xGM^. 

Dehne the value function as V (x) = inf^^-gn C{x, vr). The HJB equation for 
the problem is (cf. [7]) 

(29) Cf + H{x,Df)-jf = 0, 
where C = ( 1 / 2 ) 

H{x,p) = U) ■p + L{x,U)]. 

The equation is considered on with the growth condition 

(30) 3c,m, |/(x)| < c(l-|-llxlP), xGR'^. 

Definition 2. Let x G R^ be given. We say that a measurable function 
/i: R^ ^ U is a Markov control policy if there is an admissible system vr and a 
controlled process X corresponding to x and vr, such that Us = h{Xs), s > 0, 
P-a.s. We say that an admissible system vr is optimal for x if P (x) = C{x, vr). 
We say that a Markov control policy is optimal for x if at least one of the 
admissable systems corresponding to it is optimal. 


DIFFUSION MODEL OF SCHEDULING CONTROL 


11 


Different parts of our main result below require different assumptions on 
the abandonment rates: 

(31) y{i,j)e£, Oi<^iij, 

(32) 6i<^iij. 

These assumptions are rational for the following reason. If (31) does not 
hold [and certainly if (32) does not hold], there is a class i where customers 
leave the system by abandonment more quickly than they do by getting 
served at a certain station j ~ i. Thus under many reasonable performance 
criteria (e.g., any increasing functional of the queue lengths) it is preferable 
to never use activity (f,j). This stands in contrast to our work conservation 
assumptions. 

Theorem 1. Let Assumptions 1 and 2 hold. In addition, let one of the 
following conditions hold. 

(i) For {i,j) € T, giij depends only on i, or for {i,j) G T, depends 
only on j. In addition 9i = 0, i & I. 

(ii) The tree T is of diameter 3 at most and (31) holds. 

(hi) There are 01,02 > 0 such that L{x,U) > oiljxll™^ for all ||x|| > 02 
and all U G U (where mi is as in Assumption 2). In addition, (32) holds. 
(iv) The function L is bounded. 

Then the value V is in and it solves (29) and (30). In cases (i) and 
(ii) [resp. (hi) and (iv)] this solution is unique in Cp^j (resp. Cp^j .,.; C‘(). 
Moreover, there exists a Markov control policy that is optimal for all x G 

We end this section with a few remarks on the role of the treelike assump¬ 
tion in this work. Our results strongly depend on estimates on moments of 
the controlled process that are sub exponential in the time variable [in cases 
(i) and (ii) of the main result]. These are obtained by considering a deter¬ 
ministic model in place of (18)-(22). Use w in place oi x + W [where as in 
(18) X is the initial condition for X] and use x = x{t) (resp. y, z, fj) in place 
of X (resp. Y, Z, T). Then 





je J, 
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(36) yi-,Zj>^, i€l,jej, 

(37) e ■ y A e ■ z = 0, 

where = 0, i 9 ^ j. The first two cases of Theorem 1 are based on showing 
that (33)-(37) imply an estimate of the form 

(38) ||x(t)|| <mo(l + t)"*°(l + ||rc||D”'°, 

where niQ does not depend on t, w, y, z and V’- If tbe treelike assumption is 
removed, this estimate does not hold true in general (as shown in the example 
below). We leave open the question of whether this implication holds true 
in full generality under the treelike assumption. We stress that this is the 
reason for imposing the treelike assumption in this paper, rather than the 
using capability to rephrase the model equations (18)-(22) as (23): Although 
(23) is useful, its absence would not be a serious obstacle to treating the 
problem, whereas the large time estimates constitute a key ingredient of the 
proof. 

Example 1. Consider a system with classes 1 and 2 and stations A and 
B, and with fj,iA = y 2 A = 1, yiB = y 2 B = 2, and arbitrary 9i and 62 - Con¬ 
sider W = 0, IpiA = -'4’2A = k, ll^iB = = -^(1 + e“^*)/2, Xi = -X 2 = 

k{l — e“^*)/2, y = 0 and z = 0. Then (33)-(37) hold for every A: > 0. Thus 
(38) cannot hold. 

Finally, there is another central role played by the treelike condition, as 
elaborated in [2]. The diffusion model turns out to depend on whether pre¬ 
emption is allowed in the queueing system under scaling. As explained in 
[ 2 ], preemptive and nonpreemptive policies give rise to genuinely different 
diffusion models if the treelike assumption does not hold, whereas under 
the treelike assumption the corresponding diffusion models coincide (as sup¬ 
ported by the result in [4] for the case of a single station and in [2] for cases 
(i) and (ii) of Theorem 1). 

3. Estimating the state X in ter m s of e • X. While the relationship 
||y(t)|| -|- ||y(t)|| < c||A(t)|| is immediate from (25) and (26), the following 
result shows that in a weaker sense Y, Z and x + W dominate X (or in the 
deterministic notation, y, z and w dominate x). The result uses only the 
relationships (33)-(35), and not the further constraints (36) and (37). 

Proposition 1. Let (33)-(35) hold. Then there is a constant mi not 
depending on 'ilj,w,x,y, z or t, such that 

P'i/j{t)\\ + ||x(t)|| < mi{\\w\\l + py\\l + pz\\l), 


t>0. 
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Note that if, in addition, (36) and (37) are assumed, then ||iJy||^ = 3(e ■ 
and \\'3z\\l = J(e • x)~{t). As a result of Proposition 1, the state x is 
dominated by w and e • x in the sense 


x(t)|| < midl^lli + J|e • T|(t)). 


Lemma 1. Let w be a measurable, locally bounded function and assume 
X = w — x{s) ds. Then 



(39) 


and, in particular, if fi>0, then |x(t)| < 2 \w\l, t>0. 

Proof. Uniqueness of solutions x is standard and (39) is verified by 
substitution. □ 

Proof of Proposition 1 . Observe, by replacing Wi — Opyi with Wi, 
that without loss we can take = 0 , i Gl] therefore, 9i = 0 in the sequel. 

A node in a tree is said to be a leaf if there is exactly one edge join¬ 
ing it. Recall that the tree T has k = I + J nodes and k — 1 edges. Let 
Ti, 72 ) • • • )Zt_i = T be an increasing sequence of trees as follows. For n = 
1 ,... ,k — 2, Tn is obtained from Tn+i by deleting a leaf and the edge joining 
it. Note that Tj has exactly two nodes: one in 2 and one in J'. Let Vn denote 
the vertex set of T^. Let Vn+i = Vn+i \ Vn denote the node in Vn+i that does 
not belong to Vn- 

Denote = X n Vn and Jn = J i^Vn- We show that for n = 1 ,..., k — 1 , 


if 


Xi = Wi- ^ i G Xn, 




(40) 


^ ^ — Xi -\~ Oli, i G Xn, 


^ ' '^Pij — Pji j ^ 


then 



The implication (40) (41) is proved by induction on re. 
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Induction base: n = l. Ti has exactly two nodes, say i & T and j £ J . By 
(40), Xi = Wi — HijJipij and = jPj. Hence (41) holds. 

Induction step. Assume that (40) (41) holds for n e [1 ,k — 2 ], Let (40) 

hold for n + 1. We show that (41) holds for n + 1 in the following two cases. 


Case 1 . The leaf node Vn+i is in X. Denote i^ = Vn+i and let Jq denote 
the unique node j ~ zq in Tn+i- The validity of (40) for n + 1 implies 


(42) 


Xi = Wi- ^ pLiPifij, i £ Xn, 
j&Jr, 

'y ) — Xi cXi , i £ X ^, 

^ ^ '^ij ~ Pj ? j ^ \ {jo}; 

V’jjo ~ Pjo ~ '4’iojo I 


regarding i £Xn, j £ Jn, and 


(43) 


Xio ^10 Tiojo 1'4’iojo ) '^iojo T 


10 ■ 


regarding the node zq- By (42) and the induction assumption. 




X. 


^\t 


(44) 




iein 


<Cn( E ) + E + I^V'ioiolz 

\ieln jeJn > 


By (43), 

(4:5) Xig Wig — — ^igjgJXig. 

Applying Lemma 1 to (45) and again using (43) shows 

(46) \xio{t)\ + |0^zA^ojo(^)l <cf\wiXt + l^«iolt)- 

Combining (44) and (46) establishes the validity of (41) for n+ 1. 
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Case 2. The leaf node Vn+i is in J. Denote jo = Vn+i and let io denote 
the unique node i ~ jo in Tn+i- Assuming (40) for n + 1 implies 

Xi = Wi- IXiPlfij, i€ln\ {*o}, 

j&Jn 

Y '^ij = + iGXn\{io}, 

j&'Jn 

( 47 ) ^*0 (^*0 ~ hiojo'i^iojo) ~ 'y / hioj'i^ioj ^ 

j&Jn 

^ ^ ^*0 “I" (®jo ~ '4^iojo)l 

^ ' f’ij ~ Pj ) j ^ fpii 

ieXn 

and 

(48) Piojo Pjo' 

By (47) and the induction assumption, (44) follows. Combining (44) with 

(48) gives (41) for n + 1. 

This completes the proof that (40) implies (41) for n G [1 ,k — 1], The 
result follows on taking n = k — 1 and substituting a = —y and /3 = —z. □ 

4. An integral formula for Y and Z. Equations (33)-(35) were used in 
the previous section as substitutes for (18)-(20). They express a relation¬ 
ship between the quantities x, y, z, w and if. In this section we extract a 
relationship between y, z and w alone. This relationship, in the form of an 
integral equation, is a key element in treating Theorem l(i) and (ii). 

For a G M denote 

= f + a3f. 

Lemma 1 shows that Tc is invertible. It is easy to see that the operators Tq, 
and Ty commute. If A = (ai,..., Ok) is a hnite real-valued sequence, denote 

‘TA = ‘^ai O-'-oZa^. 

Then Ta does not depend on the order of the elements of A, but it depends 
on the multiplicity of each element. Set T 0 corresponding to A; = 0 as the 
identity map. Equations (33)-(35) imply 

(49) Y = Wi- %iyi, i G I, 

j&J 

Yifij = -zj, 
i£X 


(50) 


j^J- 
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Theorem 2. Let (49) and (50) hold. 

(i) The quantities y and z solve the integral equation 

(51) '^T:AiWi-'^T:Aryi + '^T:B^Zj = {), 

i&I i€I j&J 

where Ai and Bj are finite (possibly empty) sequences with values in {jiij : {i,j) G 
£} and, for i€T, A[ is the concatenation of Ai with 6i. 

(ii) In the special case where pLij = p.j for {i,j) G £ and 6i = 0, i gT, 
equation (51) takes the form 

(52) + 

iex j&J 

(iii) In the special case where puj = fii for {i,j) G £ and 9i = 0, i gT, 
equation (51) takes the form 

(53) '^'iMiiwi-yi) A^TMie- z) = 0, 

iei 

where Mi = {ni')i'(zx^ and M = (/ii/)j/gj. 

Remark 1. (a) Writing for the n-power 3 o • • • o of the operator 3, 

it is useful to note that the integral equation (51) can be written as 

e-w — e-y + e- z 

( 54 ) mi m\ m". 

A ^ ^ ^ CLi,n3n'Wi — ^ ^ ) ^i,n-^nyi A ^ ^ ^ ^j,n‘^nZj = 0. 

iSX n=l iai n=l jdj ri=l 

Here, Ui^n, and a'(„ are positive constants that we do not 

give in explicit form. 

(b) Recall that under (36) and (37) we have y = (e-x)^M and 2 = {e-x)~v. 

As a result, (54) expresses a relationship between the data w, the controls 
u and V, and the quantity e • x alone. 

We need some notation regarding the tree T to be used in this and the 
following sections. Fix one of the class nodes, io, as a root. (Analogous 
notation applies if we fix a station node, some jo as a root.) For k = 0,1,..., 
let level k, denoted by 1^, be the set of nodes of T at distance k from the 
root io along the edges of T (see Figure 2). Note that /q = {* 0 } and that Ik 
is empty for all k large. Let also 

Lk = loAliyj---yjlk 

be the set of nodes at distance at most k from the root and let 

Afe=LfcnT, = Lfc n 77 . 
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Note that the elements of (resp. L'^) are at even (resp. odd) distance from 
the root, not exceeding k. Let K be the largest k such that 4 is nonempty. 
For a node v at level k let B{v) {B for below) be the set of nodes u' ~ u 
at level A: + 1. For a node v at level k G [1, K] let a{v) (a for above) be the 
unique node v' ^ v at level k — 

Proof of Theorem 2. (i) Suppose we prove (51) for the case 9 i = 0, 

i G X, that is, 

(55) + = 

iex j&J 

Then, for arbitrary 9i, (51) is obtained from (55) on substituting for 
Ui. Therefore, in the sequel we set = 0, i G X, and turn to prove (55). 

We show that for A: > 1, we have 

(56) ^ -yi)+ ^ ^ "^c^Aaii) = 0, 

where , Bj and Cf are (possibly empty) sequences with values in {jiij : {i,j) G 
£}, and summation over an empty set is regarded as zero. Equation (55) fol¬ 
lows on taking k larger than K. 

We prove (56) by induction on k. 

Induction base: k = 1. By (49), 

j&h 

and by (50), 

Aoj = -^j-^Aj, j^h- 
i&h 



Fig. 2. Buffer-station tree. 
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It follows that 

Wio — Via + '^ 2 / '^2 ‘^ l ^ ioa ( i )'^ ia { i ) ~ ^ 

j(zh ifzh 

and (56) holds. 

Induction step. Assume that (56) holds for k. Using (49) and then (50), 
for i G l 2 k and j = a{i) we have 

= -^22 +Wi-Vi 


= 22 \ + 22 \+wi-Vi- 


j'(^hk+l 


i'&2k+2 


Apply ‘Z(jk on the above equation [where still i £ l 2 k and j = a{i)] to get 


(57) 


j'&hk + l 




i'^hk + 2 


Let Dk = {k^ia{i))i&hk- ^PPly to (56) and use (57) to substitute for each 
summand in the third sum in (56) to get 

0 = 22 22 








^X^^2A; + 1 


i'&hk +2 ^ 


where Dk \ {//} denotes a sequence obtained from Dk by deleting from it 
one element of value /i. This proves that (56) holds for fc + 1 and completes 
the proof of part (i). 

(ii) In the case that pij = pj and 9i = 0, by (49) and (50), 

X V'ii = Wi-yi, i£l, 

j&J 


22 ~ U' > 7 ^ 77, 

i£l 


22iwi -Vi)+ 22 = 0- 

iex jej 


hence 
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(iii) In the case that /ijj = /i, and 0i = 0, by (49) and (50), 


(58) 

= wi- yi, 

j&J 

i G T, 

(59) 

o’ 

1 

II 

jeJ- 


The result follows on applying (59)- C 

Proposition 2. Let (49) and (50) hold. Assume also that 

(60) Vi > 0, Zj >0, iel,j €j, 

(61) e ■ y{t) A e ■ z{t) = 0. 

In cases (ii) and (iii) of Theorem 2 there is a constant m 2 such that 

(62) \\ym + Mt)\\<m 2 il + tr^\\w\\:, t> 0 . 

Proof. In the case yij = y,j, (52) can be written as 
e-w — e-y + e- z+ ^ = 0 . 

By (60), Zj are positive and therefore e ■ w — e ■ y + e ■ z <0. Thus by (61), 
0 < e • z < (—e • w)~^ and, therefore, 

e ■ w < e ■ y — e ■ z < e ■ w + t\e ■ w\l. 

Since by (61), ||y|| + ||z|| = \e ■ y — e ■ z\, (62) follows. 

In the case yij = /.u, applying to (58) and by (59) [or by applying 
T.fJ to (53)], we have 

iGl 

hence, by Lemma 1 and positivity of y*, 

e-w-e-y + e- z = ^yi [ {wi{s) - yi{s)) exp{-yi{t - s)) ds < cit\\w\\l 
*ex -^0 

for some constant ci. By positivity of zj and (61) we, therefore, have 

(63) ||2:(t)|| <C2(1 + ^)||^«||^ 

By (54), the positivity of its coefficients and of yi, and (63), there is a 
constant C3 such that 

(64) ||y(t)|| <C3(l + t)''3||u;||iT 

Combining (63) and (64) establishes (62). □ 
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Corollary 1. Under the assumptions of Theorem l(i), for any m>l, 
any initial condition x G and any admissible system vr G II, 

Ef\\X{t)\r < msil + ||xl|)"^3(i + ^ > 0^ 

where m3 do not depend on x, tt and t. 

Proof. By Propositions 1 and 2, 

lk(t)|| < mi{\\w\\t + py\\t + pz\\t) 

< [mi + mim2t(l + ■ 

Thus Ef\\X{t)\p < c(l + + cjlVPlIj)"* for a constant c depending 

only on mi, m2 and m, and the result follows from standard estimates on 
the Brownian motion. □ 

5. The nonidling property. This section investigates a relationship be¬ 
tween a property defined below and referred to as the nonidling property, 
and the uniform estimate 

(65) ||x(t)|| < m4(l + + llwllt), 

where does not depend on t, w, y, z and In particular, using the 
integral equation developed in the previous section, it shows that for trees 
of diameter not exceeding 3, this property implies the uniform estimate. A 
relatively simple argument then shows that the property holds for such trees 
and the estimate follows. 

Rewrite relationships (33)-(35) as in the previous section and recall rela¬ 
tionships (36) and (37): 

( 66 ) +HiP'4!ij) = Wi-yi-ePyi, iGl, 

(67) = j&J, 

i£l 

( 68 ) yi,Zj>0, ieljej, 

(69) e-y Ae- z = 0. 

Note that x [cf. (33)] is not a part of this system of equations, but can be 
obtained from it via 

Xi = Wi-^ mP'tpij - OPyt. 

We say that the system (66)-(69) incurs no idleness on [0,T) (cf. [18]) if 
Zj{t) = 0 for t G [0, T), j £ J. 
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The NONIDLING PROPERTY. For every T > 0, if the system starts with 
u;j(0) > 0, and Wi, i G X, are strictly increasing and right-continnons on 
[0,r), then the system incurs no idleness on [0,r). 

A tree T of diameter 3 has the form depicted in Figure 3, where there are 
only two nodes io G X and jo G J that are not leaves. 

Theorem 3. Let Assumption 1 hold and assume the diameter of the 
tree X is 3. Then the nonidling property [for the system (66)-(69)] implies 
the estimate (65), where does not depend on t, w, y, z and fj. 

Lemma 2. Let the assumptions of Theorem 3 hold. Fix T > 0 and let 
{•ilj,y,z,w) be given, satisfying (66)-(69) on [0,r]. Let bounded measurable 
functions fi>0, i G X, be given. Then there is a constant that does not 
depend on T, fi,'4},y, z orw, and there exist z,n)) defined on [0,r] that 
satisfy (66)-(69), and, moreover, 

(70) e ■ y > e ■ y, e ■ z <e ■ z 


and 


(71) 


Wi = Wi + fi + rji 


where rji are nondecreasing and continuous, and 0 < rji < m^T'^i'^xlfi'lT 
on [o,r]. 

Proof. It suffices to prove the lemma in the case where all but one of 
the functions fi vanish, since the argument can then be repeated. Consider 
first the case where fi vanish for all i io (and fi^ > 0). Define Wi = Wi, 
Vi = Vi and ifij = ifij for i ^ i^. Clearly (66) holds for i ^ io- Define now 
fjigj and Zj as follows. For t G 0 := {s G [0, T]: e • 2;(s) > fi^ (s)}, let j = I + 1, 
let Aj := Zj A fi^, and set f^i^j = + Aj and Zj = Zj — Aj. Similarly, for 

j G [/ + 2,7 + J], let Aj = A (/io - A7+1-Aj_i) and f^i^j = fji^j + Aj, 



Eig. 3. Tree of diameter 3. 
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Zj = Zj — Aj. Note that for all j, Zj > 0 and that (67) is satisfied by the hat 
system. It is elementary to check that 

(72) A,(t)>0, jGj; ^A,(t) = ^, t G 0. 

j&J 

As a result, with 5{t) = 0, 

(73) + fioii) - ^(t)- 

jej jej 

For f G 0'^, set Zj = 0 and V’igj = + Aj, where now Aj = Zj, j £ J'. Then 

again (67) holds for the hat system. Also (73) holds with 6{t) = /io(t) — e • 
z{t). Note that 

(74) 0<5<fi,. 

Now that '0 is defined on [0,T], let 

Wio = + XI (^*oi - V’ioi + fJ-ioj^i'^ioj - V’ioi)) + <^ + 

jej 

and let = y^g + 6 . Then (66) holds on [0,T] for i = zq- By (73), Wi^ = 
Wio + fio + Vio: where rn^ = Y. j£j f^ioj'^i'^ioj '^ioj ) + By (72), 0 < 

V’ioj — '4’ioj A fio on 0 and by construction the same holds on 0'^. Combined 
with (74), the properties of stated in the lemma follow. Inequalities (70) 
also hold by construction. Condition (69) holds for the hat system since on 
0, y = y and by (72) e ■ z < e ■ z; and on 0'^, e - z = 0. 

Next consider the case where for some zi 7 ^ zq, /i = 0 on [0,T] for all i^ii- 
The argument is similar, but slightly more complicated because zi 9^ j for 
most j. We indicate only where the argument changes. The definition of 
fjij and Zj differs as follows. Variables A^, 5 and Zj are defined as before, 
where zq is replaced by zi. Recall that in the previous case, if: is dehned as 
= 'ijj + A. In the current case, let V’qjg = + Yj '^ioj = 'f’ioj + 

for j / jo, i>iojo = ijiojo - Hj^jo and for * ^ {0, R}- Set 

T fill jo '^iijo A ^nio) T ^ T 

Wio =Wio+J2 - V’ioi) =: Wio + yig. 

jAio 

Let iji^ = yzi + 5 and let y^ = yi for all We check that all properties 

(66)-(69) are satisfied by the hat system and that the conclusions of the 
lemma all hold, except that zDjg = zujg + fio Ayzg, where there is no guarantee 
that fjig is nondecreasing (in fact, we have assumed without loss that fio = 0). 
However, this is now corrected by applying a further modification as in the 
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previous paragraph, where now we take fi^ = Y^j^jo “ V’ioi) ]> so 

that overall fi^ + is nondecreasing and /i = 0 for all z 7^ io • □ 


Proof of Theorem 3. Fix T > 0 and let {'ip,y,z,w) be given, satisfy¬ 
ing (66)-(69) on [0,T -|- 1). Let /* be defined on [0,r -|- 1) as 

/j(t) = 1-I-max( 0, sup Wi{s)]-Wi{t) + t/T=:ai{w,T)-Wi{t)+t/T. 

\ 0<S<T-|-1 / 

Let be as in the conclusion of the lemma. Then Wi = ai{w,T) + 

t/T + rii; thus t()i(0) > 0 and wi is strictly increasing on [0,r-|- 1). Therefore, 
by the nonidling property, Zj =0 on [0,T -|- 1), J € J. Applying (54) to 
w,y,z, and using the nonnegativity of yi and of the constants a'^, 

'^yi{t) <ci{l + ty^'^\wi\T, t<T, 

i£j i&X 

where c\ does not depend on w and T. Using (70) and (71), 

e'2/(t) < C2{l + ty^^{ai{w,T) +r]i{T)) < 03(1-ht)''^r(l-h ||tc||r), t<T. 
iex 

Thus on [0,T], 

e-y<cYl + Tni + \\w\\*T). 

Using again (54), now on w,y,z and equipped with the bound on y, 

e-z<C5{l + Tni + \\w\\*Y- 


As a result, 

ll?/llr+lkllT — C6(l + Ty^{l + ||ui||y), 

where cg does not depend on T, w, ip, y and z. The result now follows from 
Proposition 1. □ 


Lemma 3. Let the assumptions of Theorem 3 hold. Assume, moreover, 
that 9i < pijf, for i £2. Then the nonidling property holds. 

Proof. It is more convenient here to work with the equivalent set of 
equations (33)-(35). Fix T > 0 and consider Wi strictly increasing, right- 
continuous on [0, T), with Wi{0) > 0. For z 7 ^ zq, by (34), ipij^ = Xi — yi, hence 
by (33), Xi = Wi + Jo[—h-ijo^iis) + — 9i)yi{s)] ds. Since the second term 

in the integrand is positive, by a standard comparison result Xi >f,i, where 
solves f,i = Wi — piijQ /q f,i{s) ds. By Lemma 1, and since Wi{0) > 0 and Wi 
is nondecreasing, we have > 0. As a result, 

(75) Xi{t)>0, zt^zq, t<T. 
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Recall that e - z = {e - x) and let r = inf{t: e • z{t) > 0} = inf{t: e • x(t) < 
0}. Arguing by contradiction, assume that t <T. By right-continuity and 
(33), 

(76) e • x(r) < 0. 

By (33) and the assumptions on tc, r > 0. On [0,t), = zj = 0, j 7^ jo- 

Applying the argument in the previous paragraph to Xi ^, 

Xioit) = Wio{t) + f i-fiiojoXiois) + {Hiojo - 0io)yio{s)] ds, t < r. 

Jo 

Hence Xi^ > C on [0, r), where now ^ fg C, and using Lemma 1, 

^(t) > Wig(t)exp(-/Jigjgt) > Wig(0)exp(-yi^j^T) > 0 , t<T. 

By (33) and since Wi^ is nondecreasing on [0,T), it follows that, in fact, 
Xioix) > 0. Along with (75), this stands in contradiction to (76). Therefore, 
e ■ z vanishes on [0,T) and by (68), so do Zi, i € X. □ 

All results of this section remain valid for trees of diameter 2, as follows on 
applying them for trees of diameter 3 and choosing Hij = 0 for appropriate 
{i,j) and 9i = 0 for appropriate i (in this section we have not used the 
assumption that fiij > 0, but only fj.ij > 0i). 

Theorem 3 and Lemma 3 imply the following. 

Corollary 2. Under the assumptions of Theorem l(ii), if (66)-(69) 
are satisfied on [0,00) and w is right-continuous, then (65) holds, with 7714 
that is independent of w, if, y, z and t. Consequently, for any m>l, any 
initial condition x € and any admissible system vr G H, 

Ef\\X{t)\r < mg{l + ||x||)”^«(l +t)™«, t > 0, 

where mg does not depend on tt, x and t. 

6. Case where cost is bounded below and summary of estimates. This 
section treats part (iii) of Theorem 1. First it is shown that there exists an 
admissible system vro under which the state process satisfies a polynomial 
growth condition in time. Then it is shown that we can consider a subset 
of n of admissible systems that, in a sense, switch to ttq after some time, 
without losing optimality. The estimates for vro then remain valid in the 
switched systems. 

Proposition 3. Let Assumption 1 hold and consider the system (33)- 
(35). Fix {io,jo) G £ such that Hi^jQ > Oi^ ■ Let y = (e- x)~^u and z = {e- x)~v, 
where u{t) = and v{t) = ej^ for all t. Then the estimate ||x(t)|| < 7717(1 + 
t)™'^||7/;||)‘ holds, where mj does not depend on w and t. 
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Proof. Let io be the root. Then Jq G B^io). It suffices to show that for 
A; = 0,1,..., if i & hk and j G hk+i, then 

\'iPa{j)j{t)\, \Xi{t)\ < Ckil + ty^WwWt. 

This claim is proved by backward induction on k. 

Induction base. The variable k is the smallest number n such that l 2 n +2 = 
0. If l 2 k+i is not empty, let j G hk+i- Then by (35), V’aOL = ~Zj = 0. Let 
i G hk- For j G B{i), = 0. Let j = a{i). Then by (33), Xi=Wi- mpiljij, 

and by (34) and the assumptions, 'ipij = Xi — yi = Xi. Thus Xi solves Xi = 
Wi — yipXi and by Lemma 1, |xi|, \il 2 ij\ < c'i\wi\* for some constant d^. 

Induction step. Assuming the claim holds for A: + 1 we show that it holds 
for k. Let j G hk+i, j 7^ jo- Then by (35) and the assumption, 'ipa{j)j = 
— '^ij- Thus by the induction assumption \'da{j)j\ < ^^(l + i)'^'=||rc||*. 

Let i ^ hk, Then by (33) and (34), the bound on 'ijij, j £ B{i), just 

obtained, and the induction assumption, 

Xi{t) =g{t) - fiia{i)'3Xi{t), 

where \g{t)\ < 03(1 + i)'^ 3 ||u;||* with some constant C 3 . Hence by Lemma 1, 
|a:^i(A)| < £ 4(1 + t)'^ 4 ||i(;||*. By (34) and the induction assumption, a similar 
bound then holds for \ipia{i)\- 

Consider now io and Jq. Write yo, zq and i/jq for Zj^ and tpiQjQ, respec¬ 
tively, with a similar convention for xq, po and Oq. By (33)-(35) and the 
induction assumption, we have 

xo = gi - yo^ijo - Oo'^yo, 

'do + g 2 = Xo - yo, 

'do + g3 = -Zo, 

where \giit)\ < 04(1 + i)'^ 4 ||t(;||* and C 4 is independent of t and w. Then xq = 
gi + 3[—yog2 — goxo + {po — Go)yo] ■ Since by assumption the last term in the 
integral is nonnegative, xq > where ^ = gi — yo'^{g 2 + €)■ Fy Lemma 1, 
|?(A)| < <?4 = £ 5(1 + A)'^ 5 ||i(;||* for all t. This establishes an appropriate lower 
bound on xq. Moreover, if yo{t) = 0, then V'o(A) > 3:o(t) — 52(A) > —54(A) — 
g 2 {t), and if 50(A) > 0, then 50(A) < e • y{t) = e • x{t) and do{t) > xo{t) — 
e ■ x{t) — 52(A) = — Xi{t) — 52(A). Thus, by the induction assumption, 
'0 o(A) > + tde\\w\\l holds for all t. Since do < ~ 53 ) a similar upper 

bound on do holds. Finally, xo(A) < | 5 i(A)| -t-zioAlV^ol* and, therefore, a similar 
bound holds for xo as well. This completes the proof by induction. □ 

Throughout the rest of this section let io and jo be as in Proposition 3. 
The following result shows that it suffices to consider only a certain subset 
of the set of admissible systems that have the following property: There is a 
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stopping time ?? on the filtration associated with vr, such that on {?? < 00 }, 
u(t) = CiQ and v{t) = for all t>i}. 

Recall that on a given admissible system tt, for every initial data x there 
is a corresponding controlled process (by Proposition A.l). 

Proposition 4. Let the assumptions of Theorem l(iii) hold. Let x G 
he given. Then there is a set of admissible systems IT C 11 such that the 
following statements are valid. 

(i) For every m>l, 

Ef\\X{t)\r<msil + tr\ vren, 

where mg depends on x,jt and m, but not on t. 

(ii) We have V{x) = V{x), where P(x) = inf^^j^ C(x,vr). 

Proof. For any vr G II and any stopping time d on vr, let be the 
admissible system obtained from vr by setting u{t) = and v{t) = for 
t>d. Given x and vr G II such that C{x, vr) < 00 and for e > 0, let 

(77) CTg = inf|t: FiJ J e~'^^L{X{s),U{s)) ds < e 

and define the stopping time 

(78) i?e = inf{t>cr£:||A(t)|| 

Note that and de depend on x and vr. Note also that 

(79) > CTg —> 00 as e ^ 0. 

For short, write vr^ for Define 

n = {vr^ : TT G n, e G (0,1)}. 

For part (i), let vr G II and e be given. Then on vr^ and on the event {d^ = 00 }, 
SRPse[ 0 ,oo) ll-^('S)|| < V ||A||*^. Moreover, by Proposition 3, on the event 
{de < 00 }, for t > (Te, 

\\X{t)\\<m^{l + t-a,r^(\\X\\:^+ sup \\W{u)-WiaM 

\ uelae,t] 

Recall that IF is a Brownian motion with drift. Hence Ef (sup^gj^^ j] ||IF(u) — 

IF('!9e)||) < ci(l + t — (Te)^i for some ci independent of e and t, and the same 
holds for expectation under Ef . Hence it suffices to prove that Ef [(|!Ai||*^)”*] 
< 00 . This follows from an easy application of Gronwall’s 
lemma, using the fact that x b{x, U) is Lipschitz uniformly in (x, U). Part 
(i) follows. 


DIFFUSION MODEL OF SCHEDULING CONTROL 


27 


By Assumption 2 (iv) and Proposition 3 , 


El 


( 80 ) 


I dt 


poo 

U,<oc / e-^^L{X{t),U{t)) 

r 

<C2-EJ" l^e<oo / e~'^\l+t-de) 

L J-de 


^{i+\\x{dM + \\wt-w^jr^ dt 


poo 

JO 


u,<oc / e-'^\i+t-^er\\wt-w^)jr^dt 


+C3E1 

:= C4£'J[1^^<oo(1 + \\X{'&e)\\y^^ ^ + (^2{^) 

:= ai{e) +Q;2(e), 


where C2, C3 and C4 do not depend on e. Conditioning on using strong 
Markov and stationary increments properties of VP, and using the fact > 
(7e, we have 

poo ___ 

a 2 {e) < c^El exp(-7C7,)Ef / X sf^WW^r^ ds 

Jo 


< C5exp(-7cre), 


where C5 < 00 by properties of Brownian motion. Therefore, by ( 79 ), 02 
converges to zero as e —> 0. 

Next we show that ai(e) ^0 as e —> 0 . Below, we sometimes write d 
for de- By definition of b [see ( 27 )], || 6 (a:, 17 )|| < C6(l + ||x||) and \\b{x,U) — 
b{y, U)\\ <C6||x — yjj, where cq does not depend on x, y and U. Thus by ( 28 ), 
for any t>d, we have on the event {i? < 00}, 

||A(t)-A(^?)|| 

<C7||VP(t)-VP(l?)|| 

+ C7 [\b{X{d),U{s)) + {b{X{s), U{s)) - biX{d),U{s)))] ds 
J'd 

< C8||lT(t) - W{d)\\ + C8(t - ^)(1 + ||X(79)||) + C8 r ||X(s) - A(i 9 )|| ds, 

J'& 

where C7 and C8 not depend on t. Hence by Gronwall’s lemma. 


\\X{t)-X{d)\\ 


( 81 ) 


< C8 


sup ||VP(s) - lT(r 9 )|| + (t-??)(l + ||^(??)||) exp(C8(t-??)). 
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Let r = inf{t > 1 }: ||X(t) || < ||X(??) ||/2} (and r = 00 on {i? = 00 }). By (78), 
||X('!9)|| > Hence by (81), assuming e is small enough, we have on < 
00 } 


P(t - 7? < 


(82) 


<p( 

sup 



<p( 

sup 






> (1 + ||X(^)||)(3-' - C8exp(c8e'/2)e'/^; 


F^ 


<P 


sup C8 ||LL(s) — VL(7 ?)|| > (6e) 

sg[i9,i9+ei/2] 


-1 


F^ 


<Cgexp{-S ^). 


Denote [5e = T^/\ [lie + Then by (77), (78) and (82), and using the lower 
bound on L{x,U), for all s small. 




U,<oc e-^^LiXis),Uis))ds 


> ciqEI 


l^e<oo / e-^^(l + ||X(s) 




ds 


L J&e J 

> cioexp(-7e'/2)^J[l^,<oo(l + (1/2)||X(4)||)™" 

>cni7-{l..<oo(l + (l/2)||X(^,)||)"^- 

X P:\re -^e> e^/^\F^,]} 

> cuS^/^E-{U,<ooil + \\X{de)\\r^e-'r^^}, 


exp{--fde){Pe 




where the constants cio, cn, C 12 > 0 do not depend on e. As a result, a;i(e) —> 
0 as e —>■ 0. Thus by (80), for every vr e H and e small enough, tt^ G IT satisfies 


(83) C(x, TT^) < C(x, vr) + a(e), 

and a{e) = ai(e) + a 2 (e) —^ 0 as e —> 0. Hence V{x) < V{x) and part (ii) of 
the result follows. □ 


The following proposition summarizes our estimates in cases (i)-(iii) of 
Theorem 1. 


Proposition 5. In cases (i)-(iii) of Theorem 1, we have: 
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(i) For any x, any admissible system vr C IT [tt G IT in ease (iii)] and 
m>l, 

E:\\Xit)\r<m,il + tr^, t>0, 

where the constant mg does not depend on t (but may depend on x, vr and 
m ). 

(ii) There is a constant mig, not depending on x, such that V{x) < 
mio(l + ||x||)"*i°. 

Proof. Item (i) follows from Corollaries 1 and 2 and Proposition 4, 
respectively. Item (ii) follows from Proposition 3 [in cases (i) and (ii) alter¬ 
natively from Corollaries 1 and 2]. □ 

APPENDIX 

Proposition A.l. Let initial data x G and an admissible system 
TT G n 6 e given. Then there exists a controlled process X associated with x 
and vr. Moreover, if X and X are controlled processes associated with x and 
vr, then X{t) = X{t), t>0, P-a.s. 

Proof. Note that (x, U) i—> b{x, U) is continuous and x i— h{x, U) is 
Lipschitz uniformly in U. Consider bm, a function that agrees with b on the 
ball m), and is uniformly Lipschitz and bounded. Then strong existence 
and uniqueness for 

Xm{t) = X + rW(t) + [ hra{Xm{s),U{s))ds, 0 < t < OO, 

Jo 

holds by Theorem 1.1.1 of [5]. Since ||A'm(t)|| < ||x|| -|-c||IT(t)|| -|-c fg ||Xm(s)|| ds, 
we have ||Am(t)|| < (||x|| -|- c||IT||i )(1 -|- e'^*) by Gronwall’s lemma. Thus let¬ 
ting Tm = inf{t: ||Am(t)|| > m}, we have ^ oo a.s. Therefore, X{t) = 
liuimXm{t) for all t defines a process that solves the equation (a strong so¬ 
lution). If X and X are both strong solutions, then for every m, they both 
agree with X^ on [0,rm]. Therefore, they agree on [0, oo) a.s. □ 


Proposition A.2. Let Assumption 1 hold. Then given ai,j3j G M, i G X, 
j G J, satisfying there exists a unique solution i^ij to the set 

of equations 


(84) 

(85) 



i G X, 

iex 



where fjij = 0 for i^ j. 


30 


R. ATAR 


Proof. We use notation from Section 4. Let zq £ iZi be the root. We show 
the following claim by backward induction on k: For k G even (resp. 

odd), if i G 4 (resp. j G Ik)-, then V’ia(i) (resp. is uniquely determined 

by (84) and (85). 

Induction base. The variable k is the largest n such that In is nonempty. 
If k is even, let i ^Ik- Then B{i) is empty and (84) implies ipiaii) = cti- The 
case k odd is similar. 

Induction step. Assume the claim holds for k. Consider the case where 
k is odd (the case k even is treated similarly). For i G Ik-i, (84) shows 
V’ia(i) = cxi — V'ij) since by the induction assumption, are 

uniquely determined for j G B{i), so is This completes the proof by 

induction and the result follows. □ 

Proof of Theorem 1. Based on Proposition 5, the proof of Theorem 
1 is similar to the proof of Theorem 2 of [4]. Since this is the main result 
of this paper, we repeat it here with modifications, mainly to accommodate 
case (hi). 

We first consider (29) on a smooth open bounded connected domain F, 
satisfying an exterior sphere condition, with boundary conditions 

(86) f{x) = V{x), xedT. 

The key is a result from [9] regarding existence of classical solutions in 
bounded domains with merely continuous boundary conditions. To use this 
result, we verify the following two conditions. 

(i) We have \H{x,p) \ < c(l + ||p||) for a; G F, where c does not depend on 
X or p. 

(ii) We have H{x,p) G C^(F x for some e G (0,1). 

Item (i) is immediate from the local boundedness of h{x,U) and L{x,U). 
Next we show that item (ii) holds. For <5 > 0, let V be such that H(y,q) > 
b{y,V)-q + L{y,V)-6. Write 

H{x,p) - H{y,q) <b{x,V) ■ p + L{x,V) -b{y,V) ■ q - L{y,V) + 6. 

Using the Holder property of L in x uniformly for (x, U) G F x U and the 
Lipschitz property of 6 in x uniformly in (x,V), 

H{x,p) - H{y,q) <c\\p-q\\ + c||p|| ||x - y|| + c\\x - y\\P + 6. 

Since 5 > 0 is arbitrary, it can be dropped. This shows that H is Holder 
continuous with exponent p, uniformly over compact subsets of F x 
Hence (ii) holds. 

Defining for {x, z,p) gT x M. x , A{x,z,p) = (l/2)r^p and B{x,z,p) = 
H{x,p) — jz, we can write (29) in divergence form as 

divA(x, /, Df) + B{x, /, Df) = 0. 
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The hypotheses of Theorem 15.19 of [9] regarding the coefficients A and B 
hold in view of (i) and (ii). Indeed, B is Holder continuous of exponent p, 
uniformly on compact subsets of T x M x Moreover, with r = 0, i^lz) = 
( 1 / 2 )minjr/, fi{z) = c(l + || 2 :||), a = 2 , 6 i = 0 and ai = 0 , we check that the 
conditions (15.59), (15.64), (15.66) and (10.23) of [9] are satisfied. Theorem 
15.19 of [9] therefore applies [with condition (15.59) instead of (15.60)]. It 
states that there exists a solution to (29) in C'^’^(r) n C'(r), satisfying the 
continuous boundary condition ( 86 ). We denote this solution by /. 

Let a; G T. Let vr be any admissible system in H and let X be the controlled 
process associated with x and tt. Let r denote the first time X hits ST. Using 
Ito’s formula for the C'^’^(M+ x T) function e“'^*/(x), in conjunction with 
the inequality 

Cf{y) + b{y, U) • Df{y) + L{y, U) - jf{y) >0, y G T, [7 G U, 


satisfied by /, we obtain 


(87) 


ft/\T 

f{x)< / e-^^L(X„t/,)ds + e-^('^")/(WAr) 

ptAr 

- e-'^^Df{Xs)-rdWs. 

Jo 


Taking expectation and then sending t —> oo, using monotone convergence 
for the first term and bounded convergence for the second term, we have 
with g(t,x) = e~AV(x ), 




e-^^L{Xs,Us) ds + e-'^^V{Xr) 


= C'r,g(x,7r). 


Taking the infimum over vr G H, we have f{x) < Vr,g(a^) = H(x), x G T, by 
Proposition 5(iv). 

To obtain the equality f = V on T, we next show there exist optimal 
Markov control policies for the control problem on T. Let 


(88) Ax,U) = b{x,U)-Df{x) + L{x,U), xGr,UGU. 

Note that is continuous on T x For each x, consider the set 7 ^ 0 of 
U G U for which 


ip{x,U) = inf ip(x,V). 

UeU 

We show that there exists a measurable selection of U^,, namely there is a 
measurable function h from (r,H(r)) to (U,S(U)) with h{x) G Uj,, x G T. 

Let Xn G r and assume lim„x„ = x G T. Let Un be any sequence such 
that Un G ■ We claim that any accumulation point of Un is in U 3 ;. For 
example, if this is not true, then by continuity of p, there is a converging 
subsequence Um, converging to U, and there is a t/ such that <5 := <p(x, U) — 
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U) > 0. Hence for all m large, (p{xm, Um) > U) + 512 > ip{xm, U) + 
(5/4, contra(iicting Um £ 

As a consequence, the assumptions of Corollary 10.3 in the Appendix 
of [6] are satisfied and it follows that there exists a measurable selection 
/i:r^U of (Ua;,xEr). 

We extend h to in a measurable way so that it takes values in U 
(but otherwise arbitrary). Clearly, x h{x, h{x)) is measurable. Consider 
the autonomous stochastic differential equation 

(89) X{t) =x+ rW{t) + [ b{Xs)ds, 

Jo 

where b{y) agrees with b{y, h{y)) on F and is set to zero off F. Then b is mea¬ 
surable and bounded on By Proposition 5.3.6 of [14], there exists a weak 
solution to this equation. That is, there exists a complete filtered probability 
space on which X is adapted and W is an /-dimensional Brownian motion, 
such that (89) holds for t > 0 a.s. On this probability space, consider the 
process Ug = h{Xs). Since X has continuous sample paths and is adapted, it 
is progressively measurable (see Proposition 1.13 of [14]) and by measurabil¬ 
ity of h, so is U. Denote by vr the admissible system thus constructed. Then 
for s<r, Us€Vx, and b{Xg,Us) ■ Df{Xg) + L{Xg,Us) = H{Xg, Df{Xs)). 
Hence 


(90) Cf{X) + b{Xg,Us)-DfiXg) + L{Xg,Us)--ff{Xg) = 0, s<t. 


A use of Ito’s formula and the convergence theorems just as before now 
shows that (87) holds with equality, and 


f{x) = E^ 



e-^^L{Xg, Us) ds + e-^^V{Xr) 


Cv,g{x,'K), 


x G F, 


with g as above. This, together with Proposition 5(iv) shows that / > VT,g = 
V on F. Summarizing, f = V on F. In particular, V E C'^’^(F) and is a 
classical solution to the HJB equation. Now F can be taken arbitrarily large, 
and this shows that V E C'^’^(M^) and that it satisfies the HJB equation 
on In view of Proposition 5(ii), it also satisfies the polynomial growth 
condition in cases (i)-(iii) of the main result; in case (iv) it is trivially a 
bounded function. As a result, there exists a classical solution to (29) in 
again denoted by /, satisfying (30) and, moreover, V = f. 

It remains to show uniqueness in the appropriate class and existence of 
optimal Markov control policies for the problem on M^. In cases (i)-(iii) [resp. 
(iv)], let / E (^^^[(M^) [resp. CKM^)] be a solution to (29). Then analogously 
to (87), we obtain 


(91) fix) < e-^^LiXs,Us) ds + e-^^fiXt) - j' e-^^DfiXg) • rdWg 


0 
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Consider first cases (i)-(iii). Taking expectation, sending t —> oo, using the 
polynomial growth of / and the moment bounds on ||Xt|j asserted in Propo¬ 
sition 5(i), we have that f{x) < C{x,tt), where vr C 11 [vr G 11 in case (hi)] 
is arbitrary. In case (iv), the same conclusion holds since / is bounded. 
Consequently, f <V on 

In cases (i) and (ii), the proof of existence of optimal Markov policies as 
well as the inequality V < f on is completely analogous to that on T, 
where we replace P with The weak existence of solutions to (89) follows 
on noting that b satisfies a linear growth condition of the form \\b{y) || < x{l + 
||y||), y € and using again Proposition 5.3.6 of [14]. Then as before, (91) 
is satished with equality, and taking expectation and using the polynomial 
growth condition of / and the moment estimates on jjXjj shows that V = f 
on W. We conclude that / is the unique solution in C'pQj(M^) that V = f, 
and that there exists a Markov control policy that is optimal for all x G . 
In case (iv) an analogous result is obtained [with uniqueness in C'^(M^)] 
using the boundedness of /. 

Finally, in case (iii) there is no guarantee that the admissible system vr 
constructed using (89) is in II and, therefore, the term e~'^^E'^f {Xt) in (91) 
(that is satisfied with equality) may not tend to zero as t —> oo. However, 
in this case we claim only uniqueness among nonnegative functions / and, 
therefore, using Ito’s formula and (90) gives 




e-^^L{Xs,Us)ds + e-'<^fiXt) 


>El 


Uo 


e-^^L{Xs,Us)ds 


and f{x) > C{x,7r) >V{x). □ 
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